DOCOHEHT BESOHE 

ED 129 925 TH 005 802 



AHTHOR 
TITLE 

INSTITUTION 
POB DATE 
NOTE 



Burkhelmer, Graham J*, Jr.; And Others 
A National Study of the Hpnard Bound Program: 
Analysis, Major Findings and Implications, 
Besearch Triangle Inst*, Durham, N«C« 

Apr 76 

26p«; Paper presented at the Annual Meeting of the 
American Educational Research Association (60th, San 
Francisco, California, April 19-23, 1976) ; For 
related documents, see ED 121 99U, ED 121 995, TB 005 
233, and TH 005 813 



EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



HF-$0.83 HC-$2.06 Plus Postage. 

Academic Achievement; College Admission; Comparative 
Analysis ; *Compensatory Education Programs ; *Data 
Analysis; Dropout Rate; *High School Students; Low 
Income Groups; ♦National Programs; Post Secondary 
Education ; ♦Program Effectiveness ; ♦Program 
Evaluation; School Holding Power; Secondary 
Education; Student Motivation; Success Factors 
♦Upward Bound 



ABSTRACT 

This paper concerns a national evaluation study of 
the upward Bound (UB) Program conducted for the D,S, Office of 
Education by the Research Triangle Institute. It presents the major 
results of the study with attention to methodological considerations 
of analysis. Two sets of analyses are discussed: (1) a set designed 
to examine the effectiveness of the DB program nationally, and (2) a 
set designed to examine differential DB project effectiveness* The 
national impact findings are quite consistent in support of the 
central theme that for student outcomes relating to postsecondary 
school entry, there are strong positive relationships with DB 
participations but for outcomes related to high school academic 
success or continuation no consistent relationships to OB ^ 
participation exist. These findings suggest that the impact of the OB 
program on student participants is primarily in the area of 
facilitating entry into education beyond high school. Given a 
national program impact, findings related to differential project 
effectiveness do not reveal any systematic program treatments that 
are related to differential project success. Rather, the relative 
success of individual projects seems more related to the 
characteristics of students recruited than to functional or 
structural characteristics of the pro jects. (Author) 
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ABSTRACT 



A National Study of the Upward Bound Program: 
Analysis, Major Findings, and Implications-^^ 

G. J. BURKHEIMER, JR, 
J. R. LEVINSOHN 
A. M. FRENCH 

Research Triangle Institute 



This paper Is the fourth In a series of four papers concerning a 

national evaluation study of the Upward Bound (UB) Program conducted for 

2/ 

the U.S. Office of Education by the Research Triangle Institute.— It 
presents the major results of the study with attention to methodological 
considerations of analysis. Two sets of analyses are discussed: (1) a set 
designed to examine the effectiveness of the UB program nationally, and 
(2) a set designed to examine differential UB project effectiveness. 

The national Impact findings are quite consistent in support of the 
central theme that for student outcomes relating to postsecondary school 
entry, there are strong positive relationships with UB partlclpatloa; but 
for outcomes related to high school academic success or contiiluatlon 
no consistent relationships to UB participation exist . These findings 
suggest that the impact of the UB program on student participants is pri-=> 
marlly in the area of facilitating entry into education beyond high school. 

Given a national program impact, findings related to differentia^ 
project effectiveness do not reveal any systematic program treatments that 
are related to differential project success. Rather » the relative success 
of individual projects seems more related to the characteristics of students 
recruited than to functional or structural characteristics of the projects. 



Paper presented at the 1976 Annual Meeting of the American Educa-* 
tlonal Research Association, San Francisco, California, 23 April 1976. 
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A National Study of the Upward Bound Program: 
Analysis, Major Findings, and Implications 

INTRODUCTION 

1/ 

This paper is the fourth in a series of four papers— concerning a 
national evaluation study of the Upward Bound (UB) Program conducted for 
the U.S. Office of Education by the Research Triangle Institute. Addressed 
in this paper are the major study results with attention to methodological 
considerations of analysis. 

A considerable analysis effort was undertaken in the (UB) evaluation 

study, and only the highlights of those analyses will be addressed in this 

paper. For the interested reader, a more complete report of analysis 

2/ 

results may be obtained from the project final reports— Results reported 
in this paper represent two logically distinct sets of analyses, one set 
conducted in a classical hypothesis testing mode, the other set in a hypo- 
thesis generating mode. 

The first set of analyses discussed in this paper relates to an eval- 
uation of the national impact of the UB program on its participants— the 
major purpose of the study as originally conceived — using the student as 
the unit of analysis. Of specific concern were student outcomes related to 
three major objectives of the UB program: 

1) Increasing the high school (HS) retention rates of participants. 

2) Increasing, among particpants, the rate of entry into postsecond- 
ary education (PSE) . 



— The other three papers (AERA Discussion No. D.19, 1976 Annual AERA 
meeting, April 19-23, 1976, San Francisco, California) are: 

Pyecha, J. N. and Berls, R. Background, Objectives, and Design of the 
National Study of the Upward Bound Program . 

Bergs ten, J. Sample Design and Data Collection Procedures: National 
Study of the Upward Bound Program . 

Koo, H. P. and Burkheimer, G. J. A National Study of the Upward Bound 
Pro}^ram; Methodological and Design Considerations . 

^ Burkheimer, G. J., Levinsohn, J. R. , Koo, H. P., and French, A. M. 
Evaluation Study of the Upward Bound Program (Volume IV of A Study of the 
National Upward Bound and Talent Search Programs . Four Volumes). Research 
Triangle Park, N.C.: Center for Educational Research and Evaluation^ 
Research Triangle Institute, April 1976. 
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3) Generating the skills and motivation necessary for success in 
education beyond high school. 

These analyses speak directly to the '"worth" of the national UB 
program, examining outcome measures for UB participants relative to analo* 
gous measures for comparison student (CS) nonpar ticipants. Such compari- 
sons can validly address UB impact when control of other relevant factors 
has been accomplished (by design or statistical adjustment) . 

The second set of analyses examines whether certain structural and 
functional characteristics of UB projects are differentially related to 
student outcomes. For this set, the analysis unit is the UB project, and 
the project variation considered in these analyses is relative variability 
(i.e., one is assessing whether particular project characteristics are 
associated with more or less successful outcomes relative to other projects)* 
Again, the comparisons are valid if control for other relevant factors has 
been accomplished. 

Certain design and sampling considerations have been discussed in 
previous papers in this series and will not be addressed here. Other pre-* 
analysis considerations are central to analysis and some limited coverage 
of these matters will be presented. Space limitations will require, however, 
a very abbreviated presentation, and the interested reader is again referred 
to the final project report for more detailed treatment. 

PRE-ANALYSIS CONSIDERATIONS 

Care was taken throughout the study to avoid analyses which would 
introduce favorable or unfavorable bias toward the UB program. Consider- 
able effort was made to Insure that the data entering analysis were consis- 
tent and error free. The sample was designed to reduce many of the possible 
differences which could exist between the UB and CS groups (and which 
could, therefore, lead to spurious group differences), and statistical 
equalization of the two groups was undertaken in an attempt to reduce 
confounding of results by possible pre-process and extra-process differences 
which were not controlled by sampling. Weight adjustments for Instrument 
and item nonresponse (or indeterminate response) were employed in critical 

5 • 
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analyses In an effort to avoid distortion of the findings due to Inconrplete 
data collection. Therefore, prior to major analytic efforts » considerable 
attexitlon was directed toward the utility and feasibility of alternative 
analysis strategies and techniques, and an extensive examination of the raw 
responses, as well as derived analysis variables, was undertaken to inves-* 
tigate the quality of the available data. Since these matters reflect 
directly on a meaningful interpretation of results of the major analyses, 
they will be briefly considered. 

Return Rates 

Return rates for various instruments, which have been presented in a 

previous paper in this series, have two obvious Implications for analysis, 

both of which are related to the extent to which data collected represents 

the potentially available data given complete return. If nonrespondents 

mirror respondents in all relevant characteristics (an assumption that is 

commonly made but rarely justified) , then response rate presents no problem 

for analysis. More likely, however, respondents differ from nonrespondents^ 

The greater the return rate, the less the nonrespondent group can influence 

the results obtained, regardless of how different they might be from respondents* 

Both possible sources of nonrepresentativeness in collected data were 

addressed: (1) individual response rate, and (2) the clustering of nonrespon-> 

dents (i.e., the distribution of nonresponse over logical units of possible 

respondents — in this case UB projects). Return rates for major student 

3/ 

Instruments were considered generally satisfactory^ and showed no serious 
difference over projects. More importantly for analyses involving differ- 
ences between UB and CS groups, return rates did not differ within these 
two groups. Given the more reasonable assumption that any bias, introduced 
as a result of nonresponse is similar for UB and CS groups, then comparisons 
of the two groups should, not reflect that bias. 



— Overall student response rate (return of one or more questionnaire) 
was quite high, 99.8 percent for the UB group and 98.3 percent for the CS 
group, reflecting extensive data collection efforts in the fall 1974 to 
reach all previous nonrespondents. Telephone follow-up during the fall 
1974 data collection period raised the fall 1974 data return rates to 
80.2 percent and 78.1 percent for the UB and CS groups, respectively. 
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Return rates and differential project response were considered less 
satisfactory for the staff Instruments, particularly In light of the Impor- 
tance of the study to these staff members. Nevertheless, project staff 
questionnaire responses were within acceptable levels. . . 

Data Quality 

While response rates to the instruments define the upper limit of data 
availability, high and noodif f erential return rates do not in themselves 
assure data integrity. Item nonresponse, careless response, and deliberately 
misleading response can greatly degrade data quality even if 100 percent 
return of questionnaires were realized. Such sources of response error are 
usually reflected as missing data, out of range data, multiple responses, 
or inconsistent data. In attempts to rectify some of these problems, 
techniques of Imputation were often employed. 

Examinations of indeterminate data suggested that the incidence of 
multiple and out of range responses was very low, accounting for no more 
than 0.2 percent of the data, on the average, for any given questionnaire 
item* The relative frequency of imputed responses was, on the average, 
even lower. The incidence of item nonresponse was considerably greater; 
however, this problem seemed to be concentrated within a few items and 
individuals. In all, the extent of unavailable data had no serious impact 
on analysis. 

The extent of logically inconsistent responses within, specific ques- 
tionnaires was, in some cases, considerably greater than would be desired* 
Proportionately large numbers of inconsistent responses were, however, 
traced to items which appeared on reexamination to be somewhat ambigious. 
Nonetheless, incidence of inconsistency still remained at upwards of five 
percent for some relatively unambigious items « Inconsistency of responses . 
between instrtiments (in most cases reflecting response differences from tw 
different reporting sources) was also observed. In light of the basically 
objective nature of the responses compared, such observed inconsistencies 
were relatively large. Although Inconsistent data were not used in analysis, 
the real danger of high rates of observed inconsistency lies not in the 
items for which the inconsistency is observed but with those items which 
were not amenable to consistency checks. A high rate of observed inconsis- 



tency is symptomatic of one or more factors which could be influencing some 
or all of the remaining data items (e.g., ambiguity of question wording, 
inattention or carelessness on the part of the respondent, or deliberate 
attempts to provide false data on the part of the respondent). The consis- 
tency checks performed signal the strong possibility of existence of one or 
more of these factors, which reduces the reliability, and thus the validity, 
of the data. 

Weight Adjustments for Nonresponse 

To provide unbiased estimates of population parameters, sampling 
weights (the inverse of the probability of selecting the student or pro- 
ject) were used in the analyses. The use of sampling weights overcomes 
possible bias in resulting statistical estimates which may be introduced 
because students or projects were selected with unequal probability (due to 
oversampling of specific groups) . 

In cases of nonresponse or other forms of indeterminate data, weight 
adjustments were made. Such adjustments involve apportioning the sampling 
weights of nonrespondents proportionally among the respondents who are in a 
category most like them (e.g., in the case of students, to those other 
students from the same project or school, and of the same sex, race, aca- 
demic risk status, poverty status, and grade level). The adjustment tech- 
nique may be expressed succinctly as follows: 

( E"ij 1 

w' = w I 1 + -iM^ ( , for respondent and 

ij ij 2 ( 

I 1 

t 

w^^ =» 0, for nonrespondents, 

where w^^ is the adjusted weight for unit j in category i, w^^ is the raw 
sampling weight for that individual, NR is the set of nonrespondents in 
category i, and R is the set of respondents in category i. In terms of 
estimates obtained, this technique yields results that are equivalent to 
those from a technique of assigning an imputed value of the category mean 
to each nonrespondent . In terms of standard error computation, however, 
the tw<5 techniques do not yield equivalent results. 
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Adjustments for Group Differences 

In order to provide more meaningful analyses of student outcomes, a 
balancing procedure was used to statistically equalize the UB and CS group 
with respect to certain pre-process and extra«-process variables. In cases 
such as this, where moderator variables (pre-process variables) are related 
to dependent variables (student outcomes) and where the distributions of 
these moderator variables are different for the groups^ under consideration 
(UB and CS) , It Is usually sound analytic practice to adjust or correct the 
dependent variables for the Influence of such moderator variables. The 
balancing procedure, which has been described In the third paper In this 
series » forced an equivalence of the UB and CS groups on the distribution 
characteristics which are related to outcomes but which were not completely 
controlled by the study design (I.e., sex, race, academic risk status, 
poverty level status, and grade In HS) , thus bringing about greater comparability 
of the two groups. 

Other Uncontrolled Sources of Group Differences 

Although the balancing procedure used all variables available for the 
entire student group and Introduced the desired statistical control for 
those variables used In the balancing. It was not expected that this would 
eliminate all group differences related to other relevant variables (differ- 
ences In Input and/or exposure to relevant processes other than UB) • Some 
Insight Into the extent to which the two groups still differed on relevant 
pre-process and extra-process variables was possible, using data which was 
available for only a subset of students (I.e., those responding to the 
Basic Student Questionnaire) • These examinations were made after balancing 
had been performed, and the results thus reflected any additional differences 
between the UB and CS groups after correction for sex, race, poverty status, 
academic risk status, and grade level (l«e. , residual differences). Vari- 
ables examined Included place of residence, highest educational attainments 
of parents, parents* occupations, previous high school course of study, 
participation in other educational intervention programs, and specific 
services received from other intervention programs. 
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There were very few statistically significant differences between the 
UB and CS group in these residual difference comparisons. Those significant 
differences uncovered wera^ in general, quite minor and the net effect of 
these input differences would be expected to result in a small bias favoring 
the CS group. There are, undoubtedly, other extra-process variables on 
which these two groups differ. Since there were no measures of these 
variables, the nature of such differences and their possible influence on 
analysis results is a matter of speculation. In general, however, the two 
groups appeared quite comparable after balancing. 



DIFFERENCES IN EDUCATIONAL CONTINUAHCE 
BETWEEN UB AND CS GROUPS 



Two of the major objectives of the UB program involve educational 
continuance: (1) continuance in and completion of HSy and (2) PSE entry. 
The extent to which the program meets these objectives nationally is the 
subject of this section. 



High School Continuance 

Table 1 presents full year HS continuance rates (completion rates for 

twelfth graders) for the UB and CS groups by grade level. In addition to 

presenting the total grade level rate for the UB group. Table 1 provides 

continuance rates within grade level by length of UB program participation. 

There were some cases for which cQntinuance could not be determined due to 

missing data. Three continuance rates were therefore computed and reported. 

These three rates are: (1) maximtim rate (P^^^) , computed by assximing that 

all indeterminate responses would have indicated continuance; (2) minimum 

rate (P . ) » computed by assuming that all Indeterminate responses would 
mxn ~~ 

have indicated noncontinuance; and (3) an adjusted rate (P computed by 

4/ 

adjusting for "item" nonresponse.— - The values of P^^jj the best avail- 
able single estimate of the continuance rates, while P and P . define 

max mm 

the upper and lower limits, respectively, of this rate for these groups of 
respondents. 
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— Differential adjustment for nonresponse due to tionselection during 
the fall 1974 subsampling was employed. ^ 
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Table 1 



FAIX 73 TO FALL 74 HIGS SCHOOL C0NTINUANCE/C0MPL2TI0M KATES BY 
GRADE PGR UPWARD SOUND PARTICIPANTS AND COMPARISOJI GROUP 







Scudent Grouo 




Grada 




UB la 
Program at 
Grad« 10 


U3 Joining 
Program In 
Grade 11 


UB Joining 
Program In 
Grada 12 


UB Total 

Grouo 


CS 


Standard 

Error^' 




0Z3X 


88. 3Z 


89. 4X 


89. 4Z 


89.3% 


91.1% 


2.2% 






OO. 24 


O / . 




oo. 4% 


ol.ol% 


3.1 


12 




216 


411 


436 


1063 


549 






p 

adj 


OQ 


07. OX 


88.9% 


88.8% 


90.1% 








214 


40 J 


421 


1040 


496 






? 


91.12 


91. 7Z 


\ / 


91.6% 


92.4% 


2.2 






83. 5Z 


85. 8Z 


\ / 

\ / 


85.3% 


80.5% 


2.5 ' 






228 


738 




966 


557 






p 

adj 


90.7% 


91. 4Z 




91.3% 


91.3% 


2.3 






210 


691 




901 


472 






p y 

max 


93. 5Z 






93.5% 


86.6% 


3.5 




OS la 


89. 5Z 






89.5% 


79.1% 


4.1 


. 10 




4U 


X 


X 


413 


571 






p s/ 

adj 


93.4% 






93.4% 


85.5% 


3.7 




si/ 


392 






392 


522 





















NOTE: Valufts raportad ara basad on tialghted data* ualag balanced CS 
weights 9 corrected for fall 1974 subsampllng and adjusted for 
cases with Indeterminate classification as to grade level or 
length of time In UB. 

^ Standard errors presented are computed for the difference In rates for the 
total group of UB participants vlthln a particular grade level. 

^ Computed by assuming all indeterminate contlnuaclon ca^es as coneinuing 
or completed. 

— ^ Computed b/ assuming all Inde terminate continuation cases as aot 
coatinuing or completing. 

— ^ Unweighted cell size in computing P^^^^ and P^jj^^* 

^ Computed by adjusting weights for indeterminate continuation/completioa 
cases . 



^ Unweighted 'ceXX sise in computing P 



adj- 
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Nine distinct subgroups are defined by the data presented In Table 1. 
Maintaining comparisons within grade level, only six Independent compari- 
sons of continuance rates among these nine groups are possible* The a priori 
comparisons decided upon were: 

1) All UB twelfth graders compared with CS twelfth graders. 

2) All UB eleventh graders compared with CS eleventh graders. 

3) UB tenth graders compared with CS tenth graders* 

4) UB twelfth graders who joined the program In grade 11 or earlier 
compared with UB twelfth graders joining the program In grade 12. 

5) UB twelfth graders who joined the program In grade 10 or earlier 
compared with UB twelfth graders joining the program In grade 11. 

6) UB eleventh graders who joined the program in grade 10 or earlier 
compared with UB eleventh graders joining the program in grade 11. 

The first three comparisons listed examine differences between the total UB 

and the CS group within a particular grade, while the remaining comparisons, 

examine differences within the UB group and a particular grade level as a 

function of length of participation. 

Standard errors reported in Table 1 are related to the differences in 

rates for the total UB and CS groups within grade level (the first three 

comparisons listed above) . As can be seen, significantly higher rates 

exist for the UB group in the tenth grade on all three estimates (^^^c' 

P ^ , azid P UB and CS continuance rate differences for eleventh and 

mln' adj 

twelfth graders are mixed (reflecting, in part, the rather large fluctuation 

of P and P , in the CS group) , but none of the differences approach 
max min o j ftr 

statistical significance. 

The three remaining comparisons (for each of the three rates) were 
made within the UB group to discover a possible relationship between length 
of program participation and HS educational continuance. No such relations- 
ships were found. The maximum absolute difference observed was 2.3 percent 
and the minimum standard error computed was 2.6 percent. 

PSE Entry 

Differences in PSE entry rates may be considered more or less inde- 
pendent of differences in HS continuance by appropriately defining a sub- 
population of students eligible for PSE entry. One such subpopulatlon of 

12 



students is made up of HS graduates* Another population of ellglbles may 

be defined In light of the current trends toward "open door" postsecondary 

Institutions for which HS graduation Is not a necessary requisite for 

admission. The second population Is thus defined as all students not In 

HS; the fozrmer group (HS graduates) Is a proper subset of the latter. PSE 

entry rates were determined for both such subgroups* 

PSE entry rates for the total subgroups of UB and CS ellglbles are 

given In Table 2. Due to a relative large percentage of Indeterminate 

response, the only rates given are those In which weight aiijustments for 

Indeterminate response were made (analogous to the P values for HS 

aaj 

continuance). For the HS graduate subgroup, the rates are further par* 
tltloned by length of participation In the UB program. Differences between 
UB and CS groups In PSE entry rates are conspicuously large. Not only Is 
the entry rate significantly greater for the UB group In a statistical 
sense. It Is also of considerable practical significance due to the absolute 
magnitude of the difference. Among HS graduates, less than half of the 
CS gr./up enter PSE as compared to almost three-fourths of the UB partici- 
pants. In the "all eligible" group (Including HS dropouts). It Is esti- 
mated that 13 of 20 UB students enter PSE, as compared to 8 of 20 In the 
balanced CS group. 

Within the UB HS graduate group, two additional Independent a priori 
comparisons were made. HS graduates who had joined UB in or prior to the 
eleventh grade continued into PSE at a 4 percent higher rate than those 
joining in the twelfth grade, but when compared to a standard error of 4.1 
percent this difference was not statistically significant. Graduates who 
had joined UB prior to the tenth grade entered PSE at an 8.9 percent higher 
rate than those Joining in the eleventh grade » which represents a statis- 
tically significant difference with an associated standard error of 3.6. 

UB participation, therefore, seems highly related to PSE entry, in 
both a qualitative and quantitative senses Program participation, regard- 
less of length of such participation, is strongly related to Increased PSE 
entry; and, among those who complete HS, PSE rate is a mono tonic nonde- 
creasing function of length of participation in the program. 

It should be pointed out that the PSE entry rates reported In Table 2 
reflect immediate PSE entry (i.e., entry into PSE within a year of gradu- 

13 
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Table 2 



POSTSECONDARY ENROLLMENT RATES FOR UPWARD BOUND 
PARTICIPANTS AND COMPARISON STUDENTS 



i — 

Population 
Considered 




Group 


Standard 
Error^' 


UB la 
Program 
In Grade 10 


UB Joining 

Program 
In Grade 11 


UB Joining 

Program 
In Grade 12 


UB Total 
Group 


CS 


High School^^ 
Graduates— 


Rate 


78. U 

191 


69-2% 
366 


68. 2Z 
343 


70. 7Z 

900 


46.7% 
413 


5.3% 


All 

Ellgibles-^ 


Rate 


X 




X 


65. IZ 

1135 


42. 5Z 
618 


4.6% 



NOTE: Reported percentages are based on weighted data, using weights balanced 
within grade level> corrected £or fall 1974 subsampling, and adjusted 
for cases with indeterminate classification as to grade level or length 
of time in UB. 



— Standard errors presented are computed for the difference in rates for the 
total group of UB participants within a particular grade level. 

— ^ Reported rate represents PSE entry during, or prior to, fall 1974 for 
students classified as high school graduates in fall 1974. 

c/ 

— Unweighted cell sizes. 

Reported rate represents PSE entry during, or prior to, fall 1974 for ail 
members of the sample classified as not in high school in fall 1974. For these 
computations, weights were also balanced over grade level. Cases with inde- 
terminate PSE entry status (less than 0.1% and 0.3%, respectively, of the UB 
and CS weighted totals) were assumed not to have entered PSE. 
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atlon or dropout). Differences In the "lag entry" rates among the CS and 
UB groups (I.e., entry Into PSE over a longer time span) could reduce the 
original advantage of the UB participants or increase it. The data of this 
study do noty however » allow examination of hypotheses regarding "lag 
entry." Furthermore, data are not available from the current study to 
assess the PSE persistence within the two groups. That is, it is possible 
that the UB participants entering PSE do not remain there at as high a rate 
as the CS PSE entrants. Were this the case, PSE completion for the two 
groups could be equivalent or even greater for the CS group. 

Types of PSE Entered 

Given the greater rate of PSE entry among UB participants, the ques- 
tion of what types of PSE institutions which were entered logically follows. 
Table 3 indicates the type of PSE institutions in which UB and CS group 
members actually enrolled. The percentages reported in Table 3 are based 
on the subset of UB and CS group members who indicated that they had attended 
some form of PSE during or prior to fall 1974. A substantially and signifi- 
cantly greater proportion of the UB subgroup had enrolled in four-year 
colleges or universities, while a significantly smaller proportion of the 
UB subgroup had enrolled in two-year colleges and certain types of other 
training schools (i.e., vocational, trade, business, etc.). 



Longitudinal Educational Continuance 

It is possible, by use of the Markov model for educational continuance 
introduced in the second paper in this series, to exiamlne long term educa- 
tional continuance for the UB and CS groups. This use of synthetic cohorts 
allows computation of HS graduation rate, given tenth grade entry, for 
various UB entry patterns. These values were computed^^ and are presented 
in Table 4. 

The information presented in Table 4 mirrors the findings previously 
reported. Probability of twelfth grade completion given tenth grade entry 
is not significantly related to UB participation, regardless of the extent 
of that participation. On the other hand, probability of PSE entry given 
tenth grade entry is significantly related to UB participation, and the 
extent of UB participation seems linearly related to an Increase in this 
probability. 



Using adjusted, rather than minimum or maximum, rates. 
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Table 3 



TYPES OF POSTSECONDARY EDUCATION INSTITUTIONS ENTERED BY 
UPWAJO) BOUND PARTICIPANTS AND COMPARISON STUDENTS 



Type of PSE Institution 


a/ 

Percentage Attending— 


Standard 
Error of 
Difference 


UB 


CS 


Vocational, trade, business > or 
other career training school 
not requiring a high school 
diploma-^ 


3,3% 


8.0% 


2.8% 


Vocational » trade » business, or 
other career training school 
requiring a high school 
diplotaar-^ 


4 •8% 


16.3% 


3.9% 


Junior college or conzmimity 
college (two year) 


17.2% 


31.3% 


5.8% 


Four year college or university 


75,7% 


45.1% 


5.2% 


Number of Cases jCunweighted) 


832 


252 





NOTE: Reported values were computed using weighted data, after 
balancing and correcting for fall 1974 subsam'Pllng on the 
subset of students classified as attending or having attended 
some PSE institution in fall 1974. 



— For purposes of these computations, indeterminate responses (ac'cbxintlng 
for eight cases in the UB group and one case in the CS group — 1.0% and 0.6% 
of the unweighted within group totals , respectively) were considered as not 
having entered. As a result of this data treatment and the fact that an 
Individual may have attended more than one of the specified types of PSE 
icistitutions (or some other type of institution), percentages are not 
constrained to add to 100% over PSE type. 

Percent attending vocational/ technical schools is closely approximated 
by adding percents for the two subclassif ications presented, since overlap 
(attendance at both types of institutions) occurred only once in the UB group 
and once in the CS group. 
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Table 4 



PROBABILITIES OF HIGH SCHOOL COMPLETION AND PSE ENTRY, 
GIVEN TENTH GRADE ENTRY, AS A FUNCTION OF DEGREE 



OF 


EXPOSURE TO 


UPWARD BOUND 














Exoosure 


to UB 




• 


3 or More 
Years 
(Entered UB 
in Grade 10 
or Earlier) 


2 Years 
(Entered UB 
in Grade 11) 


1 Year 
(Entered UB 
in Grade 12) 


No 

Exposure 
(CS) 












High School Graduation: 










Probability 


.750 


.696 


.694 


.703 


a/ 

Difference— 


+ .047 


-.007 


-.009 




Difference 


.060 


.042 


.049 




PSE Entry: 










Probability^'' 


.602 


.527 


.472 


.316 


a/ 

Difference- 


+ .286 


+.211 


+ .156 




Standard Error of 
Difference 


.052 


.051 


.049 















NOTE: Probability values reported are computed by multiplying year 

to year continuance/completion rates, adjusted for indeterminate 
classification variables and continuance/completion index. The 
results reflect balancing of the CS group. 



— Difference is CS group probability value subtracted from UB group 
probability value. Positive differences, therefore, are favorable to the 
UB group. 

For twelfth graders, the probability of transition from twelfth grade 
to college is computed and multiplied by lower grade continuance rates. 
Probability values reported can not, therefore, be exactly reconstructed 
from previously presented results. 




DIFFERENCES BETWEEN UB AND CS GROUPS ON 
FACTORS RELATED TO SUCCESS IN PSE 

The third major objective of the UB program is to provide UB studc:nts 
with the skills and motivation necessary for success in PSE. The extent to 
which the program is meeting this objective could not be directly evaluated 
due to difficulty in defining and/or measuring such variables. Success in 
meeting the objective may, however, be examined indirectly by examining 
differences between the UB and CS groups on factors related to success in 
PSE* With the exception of HS academic measures, which were theoretically 
available for all sampled students, these examinations were based on data 
from only a subset (approximately 80 percent) of the students, namely those 
participating in the spring 1974 group administrations of the Basic Student 
Questionnaire (analyses not reported here Indicated that this group was a 
biased subsample) . Further, some reported results are based on responses to 
a set of items that were included in a nested skip pattern. In such a 
pattern, the nature of the response to one of the items determines whether 
or not an individual will answer subsequent items* The number of indivi- 
duals responding to a given item will typically decrease for items that are 
more deeply nested in the skip patter n^ Thus, the number of respondents on 
which some results are based was even further reduced. 

Academic Factors 

High school academic measures are generally predictive of success in 
PSE. Specifically examined were three high school academic change measures: 

(1) change in academe grade point average from ninth grade to current grade 

(2) change in proportion of academic credits passed from ninth grade to 
current grade; and (3) change from ninth grade to current grade in propor- 
tion of credits taken that could be classified as academic. The first and 
second variables listed above are indices of changes in high school aca- 
demic achievement. It seems reasonable that a program attempting to recruit 
high risk students and to provide them with skills necessary for success in 
PSE would attempt to improve such achievement. The third variable is an 
index of course load concentration. In preparation for entry to two- and 
four-year colleges (emphasized by the UB program) it would seem reasonable 
to stress a greater concentration of courses in the academic area. 
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There was no support £rom the data for a relationship between UB 
participation and change In these academic related factors. In fact» for 
some comparisons, the results were not even in the predicted direction. 
These analyses were, however, attenuated due to very large proportions of 
indeterminate data. 

Aspirations and Expectations 

Of the many motivational aspects related to PSE success, two of the 
most obvious' are aspirations and expectations. Part of the UB program 
function is to raise participants* aspirations to attend college and to 
provide them with reasonable expectations that these aspirations will be 
met. Differences between the UB and CS groups were examined in terms of 
stated plans for entry Into PSE, desired and expected level of educational 
attainment, and desired and expected occupations. 

Results of analyses of these variables consistently favored the UB 
participants. For>^each of grades 10 through 12» UB participants planned 
PSE entry at significantly higher rates than their CS counterparts. (As 
shown previously, these plans are. In fact, realized for a large proportion 
of the UB group) . Differences between the groups In both desired and 
expected educational attainment were statistically significant, with UB 
participants both aspiring to and expecting higher levels of education. 
Further, the UB participants desired and expected to be In occupations 
requiring greater amounts of education beyond HS at significantly greater 
rates than comparison students. 

Actions Taken In Preparation for PSE 

The counseling component of many UB projects Includes direction in 
preparation for PSE. If such counseling Is effective. It would be expected 
that UB participants would take more preparatory actions than members of 
the CS group. 

For both eleventh and twelfth graders planning PSE entry within four 
years, UB participants had, on the average, taken slgnlf leant ''y greater 
nttmbers of preparatory actions (such as visits to a campus, taking admis** 
slon tests, and making formal and Informal Inquiries concerning various 
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aspects of PSE entry) than had members of the CS group. Further, UB par- 
ticipants in the twelfth grade who reported planned entry into PSE had made 
application to a significantly greater number of PSE institutions than had 
their CS cohorts. Among those who had applied^ application to four-year 
colleges was much more prevalent and application to vocational/technical 
schools more infrequent in the UB group. While acceptance rate for the UB 
applicants was higher than that for the CS applicants it was not signifi- 
cantly so. 



Availability and Adequacy of Financial Aid for PSE 

Obtaining adequate financial aid is clearly a necessary condition for 
poverty level students to attempt (much less succeed in) PSE. The UB 
program is geared for assisting participants in applying for and attaining 
adequate financial aid packages. If this function of UB is being sii.ccess- 
fully affected, then UB participants who have applied to PSE institutions 
should apply for financial aid in greater proportions than the analogous CS 
group members. Moreover » of those applying for aid, UB participants would 
be expected *to obtain adequate aid in greater relative numbers. 

Among those twelfth grade students applyijag to PSE, a much greater 
percentage of UB participants had applied for financial aid (slightly over 
half of the CS subgroup but almost 90 percent of the UB subgroup had 
applied for aid) . Of those who reported applying for financial aid, the 
proportion of UB aid applicants who reported aci^ual offer of aid was not 
significantly greater than the comparable proportion of CS aid applicants; 
however^ among those reporting offers of aid, significantly smaller percentages 
of UB participants reported offer of inadequate aid than did nonparticipants, 
and the UB participants reported an average offer oir over $500.00 per year 
more aid (primarily due to larger grants) • 

DIFFERENTIAL PROJECT EFFECTIVENESS 

Given that the UB program appears successful at a national level, a 
logical second step in analysis involves examinations of program variation, 
and specifically, whether such variation, as reflected in differences in 
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structural or functional characteristics of projects and Inputs to projects, 
Is associated with differential student outcomes* There Is every reason 
to expect that projects differ In terms of their effectiveness as measured 
by student outcomes; only the extent and the correlates of this variation 
are In question. It Is also quite logical to assxime that there are reasons 
why some projects are more effective than others. It should be pointed 
out, however, that the variability la ef'f ectiveness Is relative variation. 
The absolute magnitude of the variation or deviations of UB projects from 
some absolute base was not the major focus of these analyses. Rather, 
attempts were made to Identify the sources of whatever variation did exist. 

The analyses used to address these questions were conducted In the 
spirit of relational analysis and should be considered exploratory for two 
major reasons. First, the program characteristic or function variables are 
based upon questionnaire data that Is subject to reporting error and biases, 
and second, not enough Is known about the UB program to test well^eflned 
models of the UB process. 

Given the nature of UB, program-related processes operate on UB 
students at the project level. Hence, the chosen unit of analysis pro- 
viding the most meaningful and Interpretable analytic base was the project. 
In order to organize the data base for a project-oriented analysis, several 
preliminary procedures were necessary. In general, the approach was to 
produce a single measure at the individual project level for each attribute 
of input, process, and output considered. This was accomplished by project 
level aggregation of student data and staff data (in the latter case, 
^SS^^g^tion within and across staff category was possible) . The resulting 
manipulations produced a data base of 54 observations on a set of variables 
covering student preprocess measures (project input), project structure and 
function (project process) , *and student outcome measures (project output). 

Several criteria were used to determine the variables to be used in 
these analyses. Measures within the three major variable domains (input, 
process, and output) were selected on the basis of face validity, vari- 
ability, stability, and extent of indeterminate data. Measures were sought 
that reflected theoretically Important attributes of the UB process (e.g., 
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kind of instruction) or that were obvious candidates for outcome indices or 
for control of differences in initial selection of UB students (e.g., ninth 
grade GPA) . Additionally, the measures were examined for within-project 
and between-project variability. Measures having little within project 
variability and high between project variability were preferred. Finally, 
the rate of indeterminate aata was considered, and if this rate was high 
for a given data element, then the variable was not included. 

Several approaches were taken in examining relationships betx/een the 
three sets of variables using multiple discriminant analysis and multiple * 
regression analysis. All approaches maintained the same basic thrust: 
relating project process measures to outcome measures, while controlling 
for differences in input. The discriminant analyses followed two distinct 
but similar paths in attempts to establish relationships among the classes 
of variables. In the first approach, projects were grouped according to 
process measures and these groups were examined for differences in the set 
of student outcome measures. In the second approach, projects were clus- 
tered in terms of student outcome measures and resultant groups were examined 
for differences in process measures. Stepwise techniques were used in all 
discriminant function solutions. Regression analyses considered the outcome 
measures individually and examined the relationsips between each measured ' 
output and subsets of the process variables. Stepwise approaches as well 
as the testing of a priori rational models were used in the regression 
analyses. Control for the set of input measures was obtained in all analyses 
by forcing input variables into the various solutions. 

In all of the analyses conducted, the set of input variables showed a 
strong relationship to the output measures. Project level student pre- 
selection factors, such as proportion of academic risk students, proportion 
of poverty level students, proportion of twelfth graders, ninth grade 
academic GPA, and academic course load, were related to project level 
output measures (e.g., average change in academic GPA, HS continuance 
rate, PSE entry rate) in intuitively reasonable directions. Moreover, 
these variables were consistently major predictors of project outcomes in 
all analyses. The project process measures, however, did not enter into 
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the solutions In any cotslstent pattern or with any consistent degree of 
predictive power. In light o£ the extreme open-endedness of the stepwise 
procedures used, the lack of a consistent pattern and absence of consistent 
predlctlveness among the process variables suggests that the entry of the 
process variables Into the solutions may be an artifactual capitalization 
on sampling fluctuations In the covar lance matrices* 

The one small hint of a relationship between process and output, after 
adjustments for input differences, comes from the frequent occurrance of 
project size and emphasis of various academic year program functions in the 
several regression and discriminant models, and the notable absence in 
such models of measures of emphasis of summer program functions. Project 
size may partially reflect activity level of the academic year program, 
since the measure represents the number of students enrolled in the project 
in the fall of 1973. This pattern of findings is suggestive of a possible 
positive relationship between level of academic year program activity and 
student outcomes. There are, of course, other plausible explanations of 
this pattern. 

DISCUSSION 

The results of the national impact evaluation are quite consistent in 
support of the central theme that for outcomes concerning PSE the UB group 
differs substantially from the CS group, but for outcomes concerning HS 
academic success or completion the groups do not differ . With few excep- 
tions, this theme is pervasive, even for results that may be attenuated due 
to excessive or differential proportions of indeterminate data. 

UB participants show greater motivation for PSE entry and for occupa- 
tions requiring higher levels of education* They take more preliminary 
actions in preparation for PSE. They apply to and enter PSE institutions 
with greater relative frequency. They apply for financial aid in greater 
relative numbers and the aid packages which they are offered are more 
adequate. Although the PSE entry rate is greater for UB participants, this 
may be due more to the higher application rate, since acceptance rates 
among those who apply is no greater for the UB group. Further, although 
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the aid packages offered to UB participants are seen as more adeqxiate, 
there Is a lack of statistical evidence that UB participants who apply for 
financial aid are offered aid at any higher rate than non-participant aid 
applicants. 

The types of PSE Institution to which UB participants apply » and 
enter, in greater numbers are four-year institutions. Application to and 
entry into this type of institution is almost double the rate of the CS 
group. Given the typically greater selectivitry and cost of this type of 
institution, the higher PSE entry rates are more impressive but the larger 
financial aid packages are less impressive* 

These results cannot be attributed to differences between the UB and 
CS groups on school-specific factors, academic risk status, poverty status, 
race, or se^, since these factors were contro3J,ed either m the study 
design or statistically through the balancing proc^'dures. Other observed 
differences between the UB and CS groups for which statistical adjustment 
were not possible, could not explain differences in PSE entry rate of the 
direction, consistency, and magnitude of those reported in Table 3, since 
the observed pre-process and extra-process differences were typically quite 
small and in a direction which should have favored the CS group over the UB 
group. 

Possible bias due to nonresponse to fall 1974 data collection was 
examined, and given the small extent of this possible bias it is extremely 
unlikely that it could account for the substantial differences in PSE entry 
rate, even in the event that the bias was operating differentially for the 
UB and CS groups. There is always the possibility in survey studies, how- 
ever, that overlooked or unmeasured (and therefore uncontrolled) variables 
have confounded the results. 

Since results are based primarily on unverified student reports, another 
possible explanation of the findings would be response bias on the part of 
students. To advance this as a tenable hypothesis for group differences, 
one would have to argue for differential response bias in the two groups. 
Given the loyalty of UB participants to their program, such argiments do 
have some intuitive appeal. Some student reports were, however, subjected 
to validation, and no differential response validity between the UB and CS 
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groups was observed • In light of these findings, explanations based on 
differential response bias are more tenuous* 

Thfc.re are two remaining plausable explanations for the finding of 
higher PSE entry rate for UB participants. The first, and perhaps most 
obvious explanation, is that participation in the UB program raises the 
probability of PSE entry. The second explanation is that some unmeasured 
variable, which is highly related to PSE entry, is a major factor in 
selection of students into the program* One such variable may be high 
motivation for educational continuance beyond high school. It was observed 
that UB participants plan and expect to attend PSE in greater proportions 
and for longer periods than the CS group. This greater motivation for 
education continuance after high school should be directly related to the 
higher incidence of PSE entry in the UB group. The question remains, 
however, as to whether this motivation was affected by the UB program. 
There is simply not enough available information from this study to deter- 
mine whether the UB program was instrumental in raising participants' 
motivation to attend PSE or whether selection into the program is based to 
some extent on already extant motivation. 

The data do support an hypothesis that the UB program is facilitating 
actions taken in preparation for PSE entry, including application. Dif- 
ferences between the two groups in motivation to attend PSE were reduced to 
some, extent by restricting the analyses to the subset of students planning 
PSE entry. Because of this restriction, quantitative motivational differ- 
ences cannot provide a plausable alternative explanation of this finding, 
although qualitative differences in degree of desire to attend PSE may 
still exist. While preparatory actions and application rates are higher In 
the UB group, actual acceptance into PSE among those students who apply 
is not significantly greater for UB participants* The taajor function of 
the UB program in this area, therefore, appears to be that of aid in the 
initiation of actions leading to admission rather than aid in gaining 
admission once application has been made. It should be noted, however, 
that the UB program is working with large proportions of '*high risk" students. 
The higher application rate among UB participants would, therefore, likely 
include among the group of UB applicants greater proportions of these high 
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risk students than among the CS applicants. I£ this is the case, then even 
a similar acceptance rate £or UB participants would suggest that the UB 
program is facilitating PS£ admission among those who apply. 

These results are quite compatible with what one would expect from a 
UB project situated at a four-year college, emphasizing the importance of 
PSE entry, providing paid visits to other colleges, providing coxinseling 
and other assistance in applying for admission and financial aid, and 
having solid contacts 'with admission and financial aid officers at several 
PSE institutions, including the host institution. 

The results of the analyses of project variation resist a simple 
interpretation, but a geueral theme is reflected throughout: projc^^ct level 
outcome show relatively consistent and strong relationships to the measured 
input characteristics of the students in the project; however, they show> 
at best, weak relationships with project process measures when adjustments 
for input measure differences are made . 

The relationships between student outcomes and preprocess character** 
istics of the students are fairly intuitive and offer little useful infor- 
mation to UB program planners. The fact that a greater proportion of 
students are placed in four-year educational institutions by projects with 
proportionately fewer academic risk students is certainly not a surprising 
finding. Any project can influence the preprocess characteristics of its 
participants through its! selection criteria (for which some lattitude 
exists) , 

The lack of systematic and consistent relationships between process 
and outcome measures is more difficult to explain. It may be that no 
relationship between process and outcome measures ex5st, although such an 
interpretation is illogical. On the other hand, the lack of observed 
relationship could be attributed to many other causes: measurement error, 
inappropriate choice of variables, or inappropriate analysis models, to 
name but a few* One of the most reasonable explanations is that measure- 
ment error and relatively pervasive relationships between input and output 
jointly mask any existing relationships between process and output. If 
this is the case, then either the relationships are relatively weak, the 
measurement error is large, or both, 
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